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The introduction of the Excellence in Research for Australia scheme has heightened debate amongst research institutions over the use of 
metrics such as citations, especially given the ready availability of citation data. An analysis is presented of the citation performance of nine 
Australian universities and the Commonwealth Scientific, Industrial and Research Organisation (CSIRO) that indicates that Australian 
Institutions perform significantly better than the global average. That said, the question is raised as to whether we are setting the bar too 
low. Finally, a tentative link between citation performance and application to innovation is noted. 


The importance of measuring research 
and development (R&D) performance 

Public funding of research and development in most 
countries is seen as being of great value to the economy 
by both direct and indirect contributions. Many studies 
have estimated the benefits and while there is no clear 
cut figure, it is generally seen that the public benefit well 
outweighs the cost of publically funded R&D (Industry 
Commission, 1995). The links between public spend¬ 
ing on R&D and innovation are documented and were 
analysed in depth in two major national studies for the 
Australian Government (Batterham, 2000; Cutler, 2008). 

The Australian Bureau of Statistics in 2008 found that 
the total funding on innovation was estimated at $29b. 
Even acknowledging the benefits, given that funds are 
always limited, obvious questions arise.Are we invest¬ 
ing enough or too much? How much should be in mis¬ 


sion-oriented R&D versus unconstrained R&D? Should 
excellence be the prime determinant for government 
funding? Should research be prioritised? None of these 
questions is simple, and one of the most significant 
challenges is the measures that can be used to gauge 
the effectiveness of R&D. 

Performance measures for R&D 

Leaving defence and matters of national security aside, 
most government funded research is published ulti¬ 
mately in the open literature. It is hardly surprising 
then that measurements based on publications feature 
so strongly in funding schemes or that there are clear 
calls for funding to be related to performance (Indus¬ 
try Commission, 1995). 

Many countries take direct measurements in the form 
of research assessment schemes. Others use the indirect 
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route, e.g. the National Science Foundation in the USA, 
and the Australian Research Council and the National 
Health and Medical Research Council inAustralia rely on 
peer review assessment for selection of projects. Peer 
review itself is heavily influenced by publications. Any 
other system would be open to the criticism of insider 
knowledge being the main determinant of funding. 

It is interesting to note that at the country level, there 
are several analyses that show the relativity between 
countries and use this to justify arguments around the 
level of funding. The work of King (2004) on the sci¬ 
entific wealth of nations set a benchmark that was fol¬ 
lowed by Mashelkar (2009) in India who used a novel 
approach of rating publications per head against GDP 
per head, thereby showing the monetary advantage of 
doing research in certain countries, India in particular. 

A similar international approach at a sectoral level 
was published recently to show that in the field of 
nanoscience, Europe and the USA publish a similar 
number of papers but the citation rate for the USA is 
over twice that for Europe.The suggestion was made 
that this may be due to the higher proportion of mis¬ 
sion oriented work in the USA through the Depart¬ 
ment of Energy funded Government Laboratories 
(Roco, 2010) than in Europe. 

It is hardly surprising that excellence features in 
any appraisal as much as the number of publications. 
Excellence is seen as a key driver in and of itself in that 
research judged to be excellent is seen as more likely to 
have a greater impact and to be more likely to attract 
collaborators. Given the availability of citation analysis 
for most publications, it appears inevitable that citations 
and the resulting impact factors will feature more and 
more in the allocation of funds, despite comments high¬ 
lighting the limitations of assessment schemes (Nature 
Publishing, 2010a, b, c;Van Noorden, 2010). 

Ultimately, the most significant measure of the effec¬ 
tiveness of R&D is its impact. It has long been recog¬ 
nised that this is the most difficult measure of all. The 
challenges include the length of time between discovery 
and application; the relative contribution of translational 
work, Intellectual property protection; development 
and scale up; marketing and speed to market. 

Whether the final user is in the public domain, e.g. 
in health education or the private domain, the chain 
between R&D and innovation involves multiple steps 
and multiple players making impact measurement a 
real challenge. While impact is currently not in the 
Excellence in Research for Australia (ERA) scheme, it 
is now under trial in several universities in the UK for 


their Research Assessment Scheme (Higher Education 
Funding Council for England 2010). 

A relation between citations and 
innovation 

Given the difficulty of measuring impact, it is informa¬ 
tive to note the work of Breitzman (2001) who investi¬ 
gated the published science that underpins the prior art 
disclosed in patents. He showed that where the prior art 
involved higher citation rankings, the companies’ stock- 
price outperformed other companies by a large margin 
over a ten year period. This is one of the few examples 
of a demonstrated connection between the quality of 
R&D as measured by citation impact and innovation as 
measured by patent activity and ultimately the stock 
price of companies. Schwartz (2004) also noted the 
same trend using a similar procedure in 2004.An analy¬ 
sis along these lines for Australia could be interesting. 

Recent citation analyses of Australia’s R&D 
performance. 

Of several analyses of Australian performance at the 
institutional level, that from the Forum of European- 
Australian Science, Engineering and Technology Dis¬ 
cussion Paper (Matthews et al. 2009) shows that 
overall, Australia performs above the world average 
but that when international collaboration is involved, 
the citation impact is markedly increased (see Figure 
1 from Mathews et al, 2009). From a simple minded 
view of economic efficiency, one might argue that 
if citation impact were the primary goal, then some 
of the funding spent within Australia should be re¬ 
directed to offshore collaboration. 

The other recent analysis that looks at the totality 
of Australia’s performance is the Australian Innovation 
System Report by the Australian Department of Innova¬ 
tion, Industry, Science and Research (2010).This report 
acknowledges the difficulty of measuring innovation 
performance and focuses on R&D capacity as an essen¬ 
tial element of the national innovation system. Given 
that the Government has a target of increasing the 
number of research groups performing at world class 
levels, the proxy of performance is taken as the number 
of research fields with higher than world average cita¬ 
tion rates over the period 2004-2008. The report sug¬ 
gests Australia achieved this level in 19 of the 22 fields. 

Given that excellence can be linked to greater 
levels of international collaboration as well as a higher 
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— — * Collaborations with USA (not involving Europe) 

-Collaborations with Europe (not involving USA) 

-Collaborations involving both Europe and USA 

Australian Publications with no International Collaborations 

Figure 1 Relative citation impact of science citation index publications 1991-2005 


impact in innovation through the support of patents, 
we may well question why being “above average” is 
adequate. For a nation that publishes of order two or 
three per cent of the world's publications, one might 
argue that the target should be much higher. 

An analysis of ten institutions in Australia 
for the period January 1999 to November 
2009 

In this paper we consider ten institutions in Australia, 
chosen in terms of the highest numerical score for 
either publications or citations totalled across all 22 of 
the fields in the Thompson ISI Essential Science Indica¬ 
tors (ESI). At the outset one notes that to be classified 
in any field, an institution must have had at least one 
paper published in the ten year period that is in the 
top one per cent of cited papers in that field. 

The independent medical research institutions war¬ 
rant comment. First, ESI analyses institutions as they 
appear on the author lists.Thus, despite close connec¬ 
tion with universities, the independent institutions 
may not appear in the associated university numbers. 
Equally, as these medical research institutions tend 
to publish in a limited number of fields, they won't 
appear in this analysis as the selection of the ten insti¬ 


tutions for analysis in this paper is based on the total 
publications and citations for all twenty-two ESI fields 
for that institution. As an example, data for the Walter 
and Eliza Hall Institute in Melbourne,Australia is noted 
at the end of the paper. 

The ten institutions ranked by citation are shown 
in Table 1. As expected, this ranking is somewhat dif¬ 
ferent to the oft repeated league tables e.g. the Times 
Higher Education World University Rankings and the 
Shanghai Jiao Tong University’s Academic Ranking of 
World Universities, and is because the analysis is based 
only on citations and covers a ten year period. 

Using a scale measure such as the total number of 
publications or citations is not an absolute measure 
of quality.An institution might put out a large number 
of mediocre publications that would nevertheless 
attract a significant number of citations. Against this, 
a small institution might have a stellar performance in 
terms of citations per paper but be too small to fea¬ 
ture in this analysis. Interestingly, while there is a gra¬ 
dation in citations/paper across the ten institutions, it 
is hardly marked. 

Next let us consider where these citations sit as 
against the rest of the material indexed in the Thomson 
Essential Science Indicators (see Table 2).The rankings 
for each field are based on the number of Institutions 
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Institution 

Citations 

Papers 

Number of Fields 

Citations per 
Paper 


Number 

Rank 

Number 

Rank 

Top 10% 

Top 10 

University of Melbourne 

365,427 

97 

28,582 

65 

2 

- 

12.8 

University of Sydney 

354,109 

99 

29,847 

55 

4 

- 

11.9 

University of Queensland 

308,191 

121 

25,777 

83 

4 

- 

12.0 

UNSW 

242,937 

173 

21,249 

125 

1 

- 

11.4 

ANU 

240,425 

111 

18,394 

153 

2 

- 

13.1 

Monash University 

235,937 

178 

20,137 

131 

2 

- 

11.7 

CSIRO 

216,123 

205 

15,603 

207 

4 

3 

13.9 

UWA 

190,737 

233 

16,214 

194 

1 

- 

11.8 

University of Adelaide 

142,250 

312 

13,156 

248 

- 

- 

10.8 

University of Newcasde 

60,138 

603 

6,540 

505 

- 

- 

9.2 


Table 1: Top 10 institutions in Australia ranked on the basis of total publications and citations listed in the 

Thomson Essential Science Indicators 


that are in absolute terms in the top 10 institutions this way or are we better targeting bringing those near 

in the world as well as a column showing where the the top (say the top 25 per cent) up to the top 10 per 

ranking fits as a percentage (the top 10 per cent of all cent? 

institutions in the field, 25 per cent, 50 per cent and 
>50 per cent). 

Of the 22 fields, Australia at an 
institution level is above average in 
nineteen of the fields as stated in 
the Australian Innovation System 
Report 2010 but more interest¬ 
ingly, if we define world class as 
in the top 10 per cent, Australia 
is world class in six of the fields 
with clinical medicine and plant 
and animal science being quite 
extraordinary results. 

Equally meritorious is that CSIRO 
at the institution level is the only 
institution in Australia that is ranked 
as being in the top 10 institutions 
in the world (absolute) and does so 
in three of the fields, none of which 
involve a particularly small number 
of institutions worldwide which 
would then inflate the likelihood of 
being in the top 10. 

The relatively large number of 
institutions publishing in particu¬ 
lar fields and ranking well below 
world average performance raises 
interesting questions. Uncomforta¬ 
ble and all as the question is, should 
our limited research funds be used Table 2: Ranking of Australian Institutions by field against the rest of the world 
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Course 

Number of Institutions 

Totalfor 
world 


Top 10 

Top 

10% 

Top 

25% 

Top 

50% 

Top 

100% 

Clinical medicine 

0 

6 

9 

10 

10 

3047 

Plant & animal science 

1 

6 

7 

9 

10 

877 

Environment/ecology 

1 

2 

5 

9 

9 

541 

Geosciences 

0 

2 

3 

6 

10 

443 

Engineering 

0 

1 

9 

10 

10 

1084 

Social science 

0 

1 

6 

9 

10 

681 

Agricultural sciences 

1 

1 

5 

6 

8 

438 

Chemistry 

0 

0 

5 

8 

10 

941 

Biology and biochemistry 

0 

0 

4 

9 

9 

714 

Computer Science 

0 

0 

4 

8 

8 

335 

Materials science 

0 

0 

4 

5 

7 

631 

Psychiatry/psychology 

0 

0 

3 

7 

9 

385 

Pharmacology & toxicology 

0 

0 

3 

4 

5 

388 

Neuroscience & behaviour 

0 

0 

2 

5 

8 

458 

Microbiology 

0 

0 

2 

4 

7 

329 

Immunology 

0 

0 

1 

6 

7 

305 

Physics 

0 

0 

1 

5 

7 

681 

Molecular biology & genetics 

0 

0 

0 

2 

9 

423 

Mathematics 

0 

0 

0 

2 

5 

194 

Space science 

0 

0 

0 

1 

4 

134 
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Data on individual fields and universities 

Sitting behind the analysis in Table 2 of overall Institu¬ 
tion rankings for each of the 22 fields is the data that 
corresponds to the performance of each Institution 
in each of the 22 fields. Appendix 1 shows the rank¬ 
ing by citation for each field for each of the top ten 
institutions. Within each field, one finds different num¬ 
bers of Institutions on a worldwide basis, e g. there are 
3047 Institutions in Clinical Medicine but only 877 for 
Plant and Animal Science.To simplify the presentation 
and facilitate comparisons, the absolute rankings are 
converted to a percentage ranking. As an example, the 
University of Sydney citations for the field of Clinical 
Medicine rank 73 from 3047 institutions worldwide 
reported in the Web of Science. This is shown on the 
diagram for Clinical Medicine as Sydney, 2.4 per cent as 
the rank has been converted to a percentage. 

The data in Appendix 1 graphically confirm the 
pre-eminence of some Australian institutions. To this 
analyst, it suggests the question that if higher citations 
imply more impact and innovation, as argued above, 
should Australia concentrate more of its research fund¬ 
ing on those that are performing at the higher levels, 
e.g. the top 25 per cent of their peers in the world? 

Data for the independent medical 
research institutions 

The analysis in this paper used citations and publi¬ 
cations across all twenty-two fields of the ESI and 
then took the top ten institutions. This absolute scale 
misses the smaller but prestigious institutions such as 
the medical research institutes. Consider as an exam¬ 
ple the Walter and Eliza Hall Institute (WEHI). For a 
ten year period from January 2000, the Institute had 
55189 citations covering seven of the twenty-two 
fields, thereby just missing out on the analysis in this 
paper. On a specific field-basis, say immunology, their 
citations of 12580 put them just behind the University 
of Melbourne (15226) and Monash University (13139) 
but ahead of all other institutions in Australia. 

Most meritorious is that WEHI has 37.2 citations per 
paper for immunology against a world average of 20.4. 

Conclusion 

An argument is outlined that high rankings on cita¬ 
tions are an indicator of more effective innovation.The 
analysis of a ten year performance window for ten of 


the top publishing institutions in Australia certainly 
supports the claim that Australia’s performance is well 
above average. 

An unanswered question logically follows of how 
much better could we do and whether this entails 
focusing effort on our top performers and those that 
are close to the top. This is a topic that demands a 
deeper analysis than this paper, with its aggregation at 
an institutional level. 

Robin J Batterham is Kernot Professor of Engineering 
at the University of Melbourne, Victoria, Australia and a 
former Chief Scientist of Australia. 
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